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Abstract 

A  CMOS  LSI  chip  developed  ^by  Carnegie-Mellon  University  (Pittsburgh,  Pa.)  in  cooperation  with 
Sandia  Laboratories  (Albuquerque,  NMJfor  implementation  of  bus-level  voters  is  described.  Special 
features  based  on  experience  with  C.vmp,  a  SSI/MSI  TTL  prototype  system,  are  presented  and 
explained.  This  document  provides  all  information  necessary  to  design  triplicated  microcomputer 
systems  using  this  chip  as  the  basis  of  the  voter  circuit.  - 
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Description  of  the  Chip 

One  standard  approach  used  to  achieve  fault  tolerance  has  been  triplication  with  voting.  The 
development  of  low-cost  large  scale  integration,  with  one-board  (or  even  one-chip)  processors,  has 
encouraged  applying  triplication  at  the  processor-memory  bus  level.  The  C.vmp  project  at 
Carnegie-Mellon  University  tested  this  concept  with  a  system  built  around  triplicated  LSI-11 
processors  [Siew77,  Siew78a,  Siew78b,  Siew78c],  The  experience  gained  in  designing  and  refining  a 
SSl/MSi  implemented  voter  circuit  led  to  the  current  LSI  chip  design. 

The  voter  chip  contains  best  two-out-of-three  voters  for  four  bus  signals:  two  bidirectional  (data) 
and  two  unidirectional  (control).  Figure  1  is  a  block  diagram  of  the  unidirectional  voter  and  Figure  2  is 
a  block  diagram  of  the  bidirectional  voter.  All  four  voters  have  a  common  multiplexing  scheme  which 
allows  flexibility  in  controlling  the  actual  voting  patterns  either  statically  or  dynamically.  In  addition, 
the  two  types  of  voters  have  separate  mechanisms  for  synchronization. 

Two  sets  of  multiplexors  control  the  voting  process  itself.  The  first  selects  among  the  voted  (best 
two-out-of-three)  result  and  the  three  separate  signals,  allowing  nontriplicated  devices  on  any  of  the 
buses  to  "broadcast"  their  signals.  The  second  set  of  multiplexors  selects  separately  for  each  bus 
whether  to  use  the  voted/broadcast  signal  or  the  unmodified  signal,  allowing  operation  as  three 
independent  computers.  The  table  below  shows  the  effect  of  the  multiplexor  control  signals  on  the 
outputs  of  the  voter  chip. 

IND  BX  BY  Output  Signal 

0  0  0  Broadcast  from  bus  A 

0  0  1  Broadcast  from  bus  B 

0  1  0  Broadcast  from  bus  C 

Oil  Voted  result  from  all  three  buses 

1  -  -  Same  as  input  signal 

In  addition  to  the  common  multiplexing  control,  the  two  unidirectional  voters  also  have  provision 
for  timing  control.  Two  distinct  synchronization  issues  are  addressed.  The  first  is  that  of 
synchronous  reception  of  a  voted  bus  control  signal  by  the  three  processors.  A  latch  controlled  by 
the  chip  signal  G  is  placed  on  the  voted  signal  line  after  the  broadcast/vote  muliplexor  and  before  the 
voting/independent  multiplexor.  Thus,  if  the  processors  (or  other  bus  level  devices)  sample  the  bus 
control  signal  on  the  edge  of  a  (common)  clock,  synchronous  reception  of  the  signal  can  be  assured 
by  latching  it  for  a  suitable  interval  each  clock  period. 

The  second  synchronization  issue  is  raised  by  the  nature  of  the  bus  protocol.  The  design 
implemented  assumes  a  "level  signaling  convention"  such  as  is  used  by  the  LSI- 11  bus  in  relation  to 
the  data  lines.  By  this  is  meant  that  certain  bus  control  signals,  by  being  at  either  a  high  or  low  level, 
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indicate  the  validity  of  the  bus  data  signals.  This  is  illustrated  in  Figure  3.  A  delay  after  the  best 
two-out-of-three  voter  is  desirable  in  order  to  obtain  the  best  results  whenever  the  timings  of  the  three 
separate  devices  are  not  identical.  A  delay  allows  a  slow  module  to  still  provide  valid  data  for  voting, 
and  also  to  stay  in  step  overall  with  the  rest  of  the  system.  However,  the  delay  wouid  unnecessarily 
degrade  performance  if  the  three  devices  have  nearly  identical  timing.  To  allow  for  this,  the  delay 
circuit  must  be  bypassed  whenever  all  three  bus  control  signals  are  in  agreement. 

The  implementation  of  a  delay  circuit  in  C.vmp  using  MSI  components  consisted  of  a  0  flip-flop  and 
a  one-shot,  as  shown  in  Figure  4.  For  the  LSI  implementation,  a  short  shift  register  which  can  be 
either  set  or  cleared  asynchronously  in  all  bit  positions  was  used.  Two  levels  of  control  over  the  delay 
desired  are  provided:  clock  rate  and  number  of  clock  periods.  The  first  is  simply  the  frequency  of  the 
input  clock  (CLK),  which  can  range  from  DC  to  0.5  MHz  (2.0  MHz  at  VDD  =  +  lOv).  The  second  is 
implemented  by  a  4:1  multiplexor  which  chooses  the  maximum  number  of  clock  periods  to  wait  before 
passing  the  (delayed)  voted  signal.  Note  that  only  the  rising  edge  of  the  voted  signal  is  delayed  -  the 
falling  edge  propagates  immediately.  The  available  delays  are  shown  in  the  table  below. 

DX  DY  Maximum  Delay 

0  0  No  delay 

0  1  Delay  up  to  2  clock  periods 

1  0  Delay  up  to  3  clock  periods 

1  1  Delay  up  to  4  clock  periods 

The  special  circuit  added  to  the  bidirectional  voters  for  purposes  of  synchronization  is  a  set  of 
latches  on  the  input  signals  from  the  processors.  The  three  separate  buses  have  separate  latch 
control  signals:  GA,  GB,  and  GC.  In  each  case,  the  corresponding  signals  are  latched  when  the  latch 
control  signal  is  low,  and  are  passed  directly  through  when  the  control  signal  is  high.  This  is  useful 
for  data  signals  which  are  strobed  by  the  leading  edge  of  a  bus  control  signal  and  removed  shortly 
thereafter  without  waiting  for  any  handshaking.  This  sort  of  scheme  is  used  by  the  LSI-1 1 ,  where  the 
SYNC  signal  strobes  the  bus  address  lines  at  the  start  of  each  bus  cycle. 

The  other  special  circuit  needed  in  the  bidirectional  voters  is  the  set  of  multiplexors  which  choose 
the  direction  of  data  flow  through  the  voter.  The  chip  control  signal  DIR  selects  either  the  external  - 
memory  and  I/O  -  bus  signals  (if  low)  or  the  processor  bus  signals  (if  high)  as  input  to  the  actual  best 
two-out-of-three  voter. 

The  flexibility  offered  by  this  chip  comes  largely  from  the  amount  of  multiplexing  performed  onchip. 
Two  base  modes  of  operation,  voting  and  independent,  are  offered  to  allow  tradeoffs  between  higher 
reliability  with  fault  masking  and  higher  performance  with  distributed  processing.  In  addition, 
nontriplicated  devices  may  be  employed  even  in  voting  mode,  allowing  such  things  as  terminals  to  be 
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selectively  redundant,  possibly  with  fault  recovery  at  the  software  level.  Latches  are  provided  to 
capture  and  hold  critical  bus  signals,  and  a  variable  delay  is  provided  for  synchronization  purposes  in 
unidirectional  voters.  This  delay  can  be  varied  both  by  changing  the  basic  clock  rate  of  the  delay  shift 
register  and  by  changing  the  selection  control  inputs  to  the  following  multiplexor.  In  addition,  there  is 
no  requirement  for  signals  through  different  chips  to  use  the  same  clock  frequency  or  multiplexor 
setting.  All  of  this  adds  up  to  a  great  deal  of  freedom  for  the  designer. 
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Summary  of  Voter  Chip  Signals 


Pin 

Name  Description 

Vqd  CMOS  power:  +  5v  to  +  lOv 

(higher  voltage  results  in  higher  speed) 

vcc  TTL  power  +5v 


25 


GND  Reference  ground 


C-MU  Voter  Chip 


5 


March  10,  i960 


Control 

Pin  Pin 

Number  Name  Description 

9  INO  Selects  between  voting  mode  and  independent  mode 

0  •  Voting  mode  (0  *  GNO;  1  =  High,  depending  on  Vqq) 
1  •  Independent  mode 

37  DIP  Selects  the  direction  of  data  flow  for  bidirectional  voters 

0  •  From  external  (memory)  to  processor 

I  -  From  processor  to  external  (processor) 

35,36  BX.BV  Selects  broadcast  or  voting  mode 

00  •  Broadcast  from  bus  A 
01  -  Broadcast  from  bus  B 
10  •  Broadcast  from  bus  C 

I I  •  Voted  signal  from  ail  three  buses 

12  G  Latch  control  for  unidirectional  voters 

0  •  Latch  voted  signal 
1  •  Pass  voted  signal 

26  GA  Latch  control  for  bidirectional  voters  on  bus  A 

0  -  Latch  (bidirectional)  signals  from  processor  A 
1  •  Pass  (bidirectional)  signals  from  processor  A 

26  GB  Latch  control  for  bidirectional  voters  on  bus  B 

0  -  Latch  (bidirectional)  signals  from  processor  B 
1  -  Pass  (bidirectional)  signals  from  processor  B 

26  GC  Latch  control  for  bidirectional  voters  on  bus  C 

0  -  Latch  (bidirectional)  signals  from  processor  C 

I  -  Pass  (bidirectional)  signals  from  processor  C 

.  43  CLK  Timing  clock  waveform  for  unidirectional  voter  delays 

38,39  DX,DY  Select  delay  for  unidirectional  voters 

00  -  No  delay 

01  -  Delay  1-2  clock  periods 
10  -  Delay  2-3  clock  periods 

I I  -  Delay  3-4  clock  periods 
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Data  Paths 


Pin 

Pin 

Number 

Name 

Description 

13 

WPA 

Unidirectional  signal  W  from  bus  A 

14 

WPB 

Unidirectional  signal  W  from  bus  B 

15 

WPC 

Unidirectional  signal  W  from  bus  C 

6 

WVA 

Voted  unidirectional  signal  W  to  bus  A 

5 

WVB 

Voted  unidirectional  signal  W  to  bus  B 

4 

WVC 

Voted  unidirectional  signal  W  to  bus  C 

16 

XPA 

Unidirectional  signal  X  from  bus  A 

17 

XPB 

Unidirectional  signal  X  from  bus  B 

18 

XPC 

Unidirectional  signal  X  from  bus  C 

3 

XVA 

Voted  unidirectional  signal  X  to  bus  A 

2 

XVB 

Voted  unidirectional  signal  X  to  bus  B 

48 

XVC 

Voted  unidirectional  signal  X  to  bus  C 

19 

YPA 

Bidirectional  signal  Y  from  processor  bus  A 

21 

YPB 

Bidirectional  signal  Y  from  processor  bus  B 

23 

YPC 

Bidirectional  signal  Y  from  processor  bus  C 

20 

YEA 

Bidirectional  signal  Y  from  external  (memory)  bus  A 

22 

YEB 

Bidirectional  signal  Y  from  external  (memory)  bus  B 

24 

YEC 

Bidirectional  signal  Y  from  external  (memory)  bus  C 

46 

YVA 

Voted  bidirectional  signal  Y  to  bus  A 

45 

YVB 

Voted  bidirectional  signal  Y  to  bus  B 

44 

YVC 

Voted  bidirectional  signal  Y  to  bus  C 

27 

ZPA 

Bidirectional  signal  Z  from  processor  bus  A 

41 

ZPB 

Bidirectional  signal  Z  from  processor  bus  B 

31 

ZPC 

Bidirectional  signal  Z  from  processor  bus  C 

42 

ZEA 

Bidirectional  signal  Z  from  external  (memory)  bus  A 

40 

ZEB 

Bidirectional  signal  Z  from  external  (memory)  bus  B 

32 

ZEC 

Bidirectional  signal  Z  from  external  (memory)  bus  C 

28 

ZVA 

Voted  bidirectional  signal  Z  to  bus  A 

29 

ZVB 

Voted  bidirectional  signal  Z  to  bus  B 

30 

ZVC 

Voted  bidirectional  signal  Z  to  bus  C 

C-MU  Voter  Chip 


7 


March  10, 1980 


Acknowledgements 

We  gratefully  acknowledge  the  cooperation  of  Chuck  Gwyn  and  Carl  Longerot  of  Sandia 
Laboratories  in  the  implementation  of  this  chip.  Without  the  efforts  of  Carl  Longerot  in  pushing  the 
initial  layout  through  to  the  final  masks,  the  chip  could  never  have  been  built. 


References 


[Siew77] 

D.  P.  Siewiorek,  M.  Canepa,  and  S.  Clark. 

C. vmp:  The  Architecture  and  Implementation  of  a  Fault  Tolerant  Multiprocessor. 
Proceedings  of  the  Seventh  Annual  International  Symposium  on  Fault-Tolerant  Computing, 

pages  37-43. 

IEEE  Computer  Society,  1977. 

[Siew78a] 

D.  P.  Siewiorek,  V.  Kini,  H.  Mashburn,  S.  R.  McConnel,  and  M.  Tsao. 

A  Case  Study  of  C.mmp,  Cm*,  and  C.vmp:  Part  l--Experiences  with  Fault  Tolerance  in 
Multiprocessor  Systems. 

Proceedings  of  the  IEEE  66(1 0):1 178-1 199,  October  1978. 

Also  available  in  Carnegie-Mellon  University  Technical  Report  CMU-CS-78-143. 

[SieW78b] 

D.  P.  Siewiorek,  V.  Kini,  R.  Joobbani,  and  H.  Beilis. 

A  Case  Study  of  C.mmp,  Cm*,  and  C.vmp:  Part  ll-Predicting  and  Calibrating  Reliability  of 
Multiprocessor  Systems. 

Proceedings  of  the  IEEE  66(10):1200-1220,  October  1978. 

Also  available  in  Carnegie-Mellon  University  Technical  Report  CMU-CS-78-143. 

[Siew78c] 

D.  P.  Siewiorek  and  S.  R.  McConnel. 

C.vmp:  The  Implementation,  Performance,  and  Reliability  of  a  Fault  Tolerant  Multiprocessor. 
3rd  USA-Japan  Computer  Conference  Proceedings,  pages  325-330. 

AFIPS  and  IPSJ,  1978. 


FIGURE  1.  UNIDIRECTIONAL  VOTER  CIRCUIT 
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FIGURE  2.  BIDIRECTIONAL  VOTER  CIRCUIT 


FIGURE  4A.  SYNCHRONIZING  VOTER 


FIGURE  4B.  DATA  VOTER 
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